Redundant memory access for rows or columns containing faulty memory cells in analog neural memory in deep learning artificial neural network

ABSTRACT

Numerous embodiments are disclosed for accessing redundant non-volatile memory cells in place of one or more rows or columns containing one or more faulty non-volatile memory cells during a program, erase, read, or neural read operation in an analog neural memory system used in a deep learning artificial neural network.

PRIORITY CLAIM

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/696,778, filed on Jul. 11, 2018, and titled, “Redundant MemoryAccess for Rows or Columns Containing Faulty Memory Cells in AnalogNeuro Memory in Deep Learning Artificial Neural Network,” which isincorporated by reference herein.

FIELD OF THE INVENTION

Numerous embodiments are disclosed for accessing redundant non-volatilememory cells in place of one or more rows or columns containing one ormore faulty non-volatile memory cells during a program, erase, read, orneural read operation in an analog neural memory system used in a deeplearning artificial neural network.

BACKGROUND OF THE INVENTION

Artificial neural networks mimic biological neural networks (e.g., thecentral nervous systems of animals, in particular the brain) which areused to estimate or approximate functions that can depend on a largenumber of inputs and are generally unknown. Artificial neural networksgenerally include layers of interconnected “neurons” which exchangemessages between each other.

FIG. 1 illustrates an artificial neural network 100, where the circlesrepresent the inputs or layers of neurons. The connections (calledsynapses) are represented by arrows, and have numeric weights that canbe tuned based on experience. This makes neural networks adaptive toinputs and capable of learning. Typically, neural networks include alayer of multiple inputs. There are typically one or more intermediatelayers of neurons, and an output layer of neurons that provide theoutput of the neural network. The neurons at each level individually orcollectively make a decision based on the received data from thesynapses.

One of the major challenges in the development of artificial neuralnetworks for high-performance information processing is a lack ofadequate hardware technology. Indeed, practical neural networks rely ona very large number of synapses, enabling high connectivity betweenneurons, i.e. a very high computational parallelism. In principle, suchcomplexity can be achieved with digital supercomputers or specializedgraphics processing unit clusters. However, in addition to high cost,these approaches also suffer from mediocre energy efficiency as comparedto biological networks, which consume much less energy primarily becausethey perform low-precision analog computation. CMOS analog circuits havebeen used for artificial neural networks, but most CMOS-implementedsynapses have been too bulky given the high number of neurons andsynapses.

Applicant previously disclosed an artificial (analog) neural networkthat utilizes one or more non-volatile memory arrays as the synapses inU.S. patent application Ser. No. 15/594,439, which is incorporated byreference. The non-volatile memory arrays operate as analog neuromorphicmemory. The neural network device includes a first plurality of synapsesconfigured to receive a first plurality of inputs and to generatetherefrom a first plurality of outputs, and a first plurality of neuronsconfigured to receive the first plurality of outputs. The firstplurality of synapses includes a plurality of memory cells, wherein eachof the memory cells includes spaced apart source and drain regionsformed in a semiconductor substrate with a channel region extendingthere between, a floating gate disposed over and insulated from a firstportion of the channel region and a non-floating gate disposed over andinsulated from a second portion of the channel region. Each of theplurality of memory cells is configured to store a weight valuecorresponding to a number of electrons on the floating gate. Theplurality of memory cells is configured to multiply the first pluralityof inputs by the stored weight values to generate the first plurality ofoutputs.

Each non-volatile memory cells used in the analog neuromorphic memorysystem must be erased and programmed to hold a very specific and preciseamount of charge in the floating gate. For example, each floating gatemust hold one of N different values, where N is the number of differentweights that can be indicated by each cell. Examples of N include 16,32, and 64.

One unique characteristic of analog neuromorphic memory systems is thatthe system must support two different types of read operations. In anormal read operation, an individual memory cell is read as inconventional memory systems. However, in a neural read operation, theentire array of memory cells is read at one time, where each bit linewill output a current that is the sum of all currents from the memorycells connected to that bit line.

Supporting both types of read operations leads to several challenges.One challenge is how to provide redundancy for the system. Wheremultiple redundant rows or columns are being used (due to the occurrenceof multiple faulty rows or columns), the system must be able to activateall of the redundant rows or columns at one time during a neural readoperation. However, in conventional systems, a read or program operationwill operate on only one row or a sector of rows at any given time—andnot all of them—and therefore only some of the redundant rows or columnswill need to be asserted at any given time. Thus, prior art decodingsystems do not support a neural read operation.

What is needed is an improved decoding system to be used with analogneuromorphic memory to provide redundancy during programming, erase,read, and neural read operations.

SUMMARY OF THE INVENTION

Numerous embodiments are disclosed for accessing redundant non-volatilememory cells in place of one or more rows or columns containing one ormore faulty non-volatile memory cells during a program, erase, read, orneural read operation in an analog neural memory system used in a deeplearning artificial neural network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram that illustrates a prior art artificial neuralnetwork.

FIG. 2 is a cross-sectional side view of a conventional 2-gatenon-volatile memory cell.

FIG. 3 is a cross-sectional side view of a conventional 4-gatenon-volatile memory cell.

FIG. 4 is a side cross-sectional side view of conventional 3-gatenon-volatile memory cell.

FIG. 5 is a cross-sectional side view of another conventional 2-gatenon-volatile memory cell.

FIG. 6 is a diagram illustrating the different levels of an exemplaryartificial neural network utilizing a non-volatile memory array.

FIG. 7 is a block diagram illustrating a vector multiplier matrix.

FIG. 8 is a block diagram illustrating various levels of a vectormultiplier matrix.

FIG. 9 depicts another embodiment of a vector multiplier matrix.

FIG. 10 depicts another embodiment of a vector multiplier matrix.

FIG. 11 depicts operating voltages to perform operations on the vectormultiplier matrix of FIG. 10 .

FIG. 12 depicts another embodiment of a vector multiplier matrix.

FIG. 13 depicts operating voltages to perform operations on the vectormultiplier matrix of FIG. 12 .

FIG. 14 depicts another embodiment of a vector multiplier matrix.

FIG. 15 depicts operating voltages to perform operations on the vectormultiplier matrix of FIG. 14 .

FIG. 16 depicts another embodiment of a vector multiplier matrix.

FIG. 17 depicts operating voltages to perform operations on the vectormultiplier matrix of FIG. 16 .

FIG. 18 depicts a memory system comprising a vector multiplier matrixand a redundancy system.

FIG. 19 depicts a prior art redundancy system.

FIG. 20 depicts an embodiment of a redundancy system for a vectormultiplier matrix.

FIG. 21 depicts another embodiment of a redundancy system for a vectormultiplier matrix.

FIG. 22 depicts another embodiment of a redundancy system for a vectormultiplier matrix.

FIG. 23 depicts another embodiment of a redundancy system for a vectormultiplier matrix.

FIG. 24 depicts an embodiment of a method of accessing redundant memorycells in place of faulty memory cells during a program, read, or eraseoperation.

FIG. 25 depicts an embodiment of a method of accessing redundant memorycells in place of faulty memory cells during a neural read operation.

FIG. 26 depicts an embodiment of a decoding system for use with a vectormultiplier matrix.

FIG. 27 depicts another embodiment of a decoding system for use with avector multiplier matrix.

FIG. 28 depicts a prior art long short term memory system.

FIG. 29 depicts an exemplary cell in a prior art long short term memorysystem.

FIG. 30 depicts an implementation of the exemplary cell in the longshort term memory system of FIG. 29 .

FIG. 31 depicts a prior art gated recurrent unit system.

FIG. 32 depicts an exemplary cell in a prior art gated recurrent unitsystem.

FIG. 33 depicts an implementation of the exemplary cell in the gatedrecurrent unit system of FIG. 32 .

DETAILED DESCRIPTION OF THE INVENTION

The artificial neural networks of the present invention utilize acombination of CMOS technology and non-volatile memory arrays.

Non-Volatile Memory Cells

Digital non-volatile memories are well known. For example, U.S. Pat. No.5,029,130 (“the '130 patent”) discloses an array of split gatenon-volatile memory cells, and is incorporated herein by reference forall purposes. Such a memory cell is shown in FIG. 2 . Each memory cell210 includes source region 14 and drain region 16 formed in asemiconductor substrate 12, with a channel region 18 there between. Afloating gate 20 is formed over and insulated from (and controls theconductivity of) a first portion of the channel region 18, and over aportion of the source region 16. A word line terminal 22 (which istypically coupled to a word line) has a first portion that is disposedover and insulated from (and controls the conductivity of) a secondportion of the channel region 18, and a second portion that extends upand over the floating gate 20. The floating gate 20 and word lineterminal 22 are insulated from the substrate 12 by a gate oxide. Bitline24 is coupled to drain region 16.

Memory cell 210 is erased (where electrons are removed from the floatinggate) by placing a high positive voltage on the word line terminal 22,which causes electrons on the floating gate 20 to tunnel through theintermediate insulation from the floating gate 20 to the word lineterminal 22 via Fowler-Nordheim tunneling.

Memory cell 210 is programmed (where electrons are placed on thefloating gate) by placing a positive voltage on the word line terminal22, and a positive voltage on the source 16. Electron current will flowfrom the source 16 towards the drain 14. The electrons will accelerateand become heated when they reach the gap between the word line terminal22 and the floating gate 20. Some of the heated electrons will beinjected through the gate oxide 26 onto the floating gate 20 due to theattractive electrostatic force from the floating gate 20.

Memory cell 210 is read by placing positive read voltages on the drain14 and word line terminal 22 (which turns on the channel region underthe word line terminal). If the floating gate 20 is positively charged(i.e. erased of electrons and positively coupled to the drain 16), thenthe portion of the channel region under the floating gate 20 is turnedon as well, and current will flow across the channel region 18, which issensed as the erased or “1” state. If the floating gate 20 is negativelycharged (i.e. programmed with electrons), then the portion of thechannel region under the floating gate 20 is mostly or entirely turnedoff, and current will not flow (or there will be little flow) across thechannel region 18, which is sensed as the programmed or “0” state.

Table No. 1 depicts typical voltage ranges that can be applied to theterminals of memory cell 210 for performing read, erase, and programoperations:

TABLE NO. 1 Operation of Flash Memory Cell 210 of FIG. 2 WL BL SL Read2-3 V 0.6-2 V 0 V Erase ~11-13 V 0 V 0 V Program 1-2 V 1-3 μA 9-10 V

Other split gate memory cell configurations are known. For example, FIG.3 depicts four-gate memory cell 310 comprising source region 14, drainregion 16, floating gate 20 over a first portion of channel region 18, aselect gate 28 (typically coupled to a word line) over a second portionof the channel region 18, a control gate 22 over the floating gate 20,and an erase gate 30 over the source region 14. This configuration isdescribed in U.S. Pat. No. 6,747,310, which is incorporated herein byreference for all purposes). Here, all gates are non-floating gatesexcept floating gate 20, meaning that they are electrically connected orconnectable to a voltage source. Programming is shown by heatedelectrons from the channel region 18 injecting themselves onto thefloating gate 20. Erasing is shown by electrons tunneling from thefloating gate 20 to the erase gate 30.

Table No. 2 depicts typical voltage ranges that can be applied to theterminals of memory cell 310 for performing read, erase, and programoperations:

TABLE NO. 2 Operation of Flash Memory Cell 310 of FIG. 3 WL/SG BL CG EGSL Read 1.0-2 V 0.6-2 V 0-2.6 V 0-2.6 V 0 V Erase −0.5 V/0 V   0 V 0V/−8 V  8-12 V 0 V Program    1 V   1 μA  8-11 V 4.5-9 V 4.5-5 V  

FIG. 4 depicts split gate three-gate memory cell 410. Memory cell 410 isidentical to the memory cell 310 of FIG. 3 except that memory cell 410does not have a separate control gate. The erase operation (erasingthrough erase gate) and read operation are similar to that of the FIG. 3except there is no control gate bias. The programming operation also isdone without the control gate bias, hence the program voltage on thesource line is higher to compensate for lack of control gate bias.

Table No. 3 depicts typical voltage ranges that can be applied to theterminals of memory cell 410 for performing read, erase, and programoperations:

TABLE NO. 3 Operation of Flash Memory Cell 410 of FIG. 4 WL/SG BL EG SLRead 0.7-2.2 V 0.6-2 V 0-2.6 V 0 V Erase −0.5 V/0 V   0 V  11.5 V 0 VProgram   1 V   2-3 μA   4.5 V 7-9 V  

FIG. 5 depicts stacked gate memory cell 510. Memory cell 510 is similarto memory cell 210 of FIG. 2 , except floating gate 20 extends over theentire channel region 18, and control gate 22 extends over floating gate20, separated by an insulating layer. The erase, programming, and readoperations operate in a similar manner to that described previously formemory cell 210.

Table No. 4 depicts typical voltage ranges that can be applied to theterminals of memory cell 510 for performing read, erase, and programoperations:

TABLE NO. 4 Operation of Flash Memory Cell 510 of FIG. 5 CG BL SL P-subRead  2-5 V 0.6-2 V 0 V 0 V Erase −8 to −10 V/0 V FLT FLT 8-10 V/15-20 VProgram 8-12 V   3-5 V 0 V 0 V

In order to utilize the memory arrays comprising one of the types ofnon-volatile memory cells described above in an artificial neuralnetwork, two modifications are made. First, the lines are configured sothat each memory cell can be individually programmed, erased, and readwithout adversely affecting the memory state of other memory cells inthe array, as further explained below. Second, continuous (analog)programming of the memory cells is provided.

Specifically, the memory state (i.e. charge on the floating gate) ofeach memory cells in the array can be continuously changed from a fullyerased state to a fully programmed state, independently and with minimaldisturbance of other memory cells. In another embodiment, the memorystate (i.e., charge on the floating gate) of each memory cell in thearray can be continuously changed from a fully programmed state to afully erased state, and vice-versa, independently and with minimaldisturbance of other memory cells. This means the cell storage is analogor at the very least can store one of many discrete values (such as 16or 64 different values), which allows for very precise and individualtuning of all the cells in the memory array, and which makes the memoryarray ideal for storing and making fine tuning adjustments to thesynapsis weights of the neural network.

Neural Networks Employing Non-Volatile Memory Cell Arrays

FIG. 6 conceptually illustrates a non-limiting example of a neuralnetwork utilizing a non-volatile memory array. This example uses thenon-volatile memory array neural net for a facial recognitionapplication, but any other appropriate application could be implementedusing a non-volatile memory array based neural network.

S0 is the input, which for this example is a 32×32 pixel RGB image with5 bit precision (i.e. three 32×32 pixel arrays, one for each color R, Gand B, each pixel being 5 bit precision). The synapses CB1 going from S0to C1 have both different sets of weights and shared weights, and scanthe input image with 3×3 pixel overlapping filters (kernel), shiftingthe filter by 1 pixel (or more than 1 pixel as dictated by the model).Specifically, values for 9 pixels in a 3×3 portion of the image (i.e.,referred to as a filter or kernel) are provided to the synapses CB1,whereby these 9 input values are multiplied by the appropriate weightsand, after summing the outputs of that multiplication, a single outputvalue is determined and provided by a first neuron of CB1 for generatinga pixel of one of the layers of feature map C1. The 3×3 filter is thenshifted one pixel to the right (i.e., adding the column of three pixelson the right, and dropping the column of three pixels on the left),whereby the 9 pixel values in this newly positioned filter are providedto the synapses CB1, whereby they are multiplied by the same weights anda second single output value is determined by the associated neuron.This process is continued until the 3×3 filter scans across the entire32×32 pixel image, for all three colors and for all bits (precisionvalues). The process is then repeated using different sets of weights togenerate a different feature map of C1, until all the features maps oflayer C1 have been calculated.

At C1, in the present example, there are 16 feature maps, with 30×30pixels each. Each pixel is a new feature pixel extracted frommultiplying the inputs and kernel, and therefore each feature map is atwo dimensional array, and thus in this example the synapses CB1constitutes 16 layers of two dimensional arrays (keeping in mind thatthe neuron layers and arrays referenced herein are logicalrelationships, not necessarily physical relationships—i.e., the arraysare not necessarily oriented in physical two dimensional arrays). Eachof the 16 feature maps is generated by one of sixteen different sets ofsynapse weights applied to the filter scans. The C1 feature maps couldall be directed to different aspects of the same image feature, such asboundary identification. For example, the first map (generated using afirst weight set, shared for all scans used to generate this first map)could identify circular edges, the second map (generated using a secondweight set different from the first weight set) could identifyrectangular edges, or the aspect ratio of certain features, and so on.

An activation function P1 (pooling) is applied before going from C1 toS1, which pools values from consecutive, non-overlapping 2×2 regions ineach feature map. The purpose of the pooling stage is to average out thenearby location (or a max function can also be used), to reduce thedependence of the edge location for example and to reduce the data sizebefore going to the next stage. At S1, there are 16 15×15 feature maps(i.e., sixteen different arrays of 15×15 pixels each). The synapses andassociated neurons in CB2 going from S1 to C2 scan maps in S1 with 4×4filters, with a filter shift of 1 pixel. At C2, there are 22 12×12feature maps. An activation function P2 (pooling) is applied beforegoing from C2 to S2, which pools values from consecutive non-overlapping2×2 regions in each feature map. At S2, there are 22 6×6 feature maps.An activation function is applied at the synapses CB3 going from S2 toC3, where every neuron in C3 connects to every map in S2. At C3, thereare 64 neurons. The synapses CB4 going from C3 to the output S3 fullyconnects S3 to C3. The output at S3 includes 10 neurons, where thehighest output neuron determines the class. This output could, forexample, be indicative of an identification or classification of thecontents of the original image.

Each level of synapses is implemented using an array, or a portion of anarray, of non-volatile memory cells. FIG. 7 is a block diagram of thevector-by-matrix multiplication (VMM) array that includes thenon-volatile memory cells, and is utilized as the synapses between aninput layer and the next layer. Specifically, the VMM 32 includes anarray of non-volatile memory cells 33, erase gate and word line gatedecoder 34, control gate decoder 35, bit line decoder 36 and source linedecoder 37, which decode the inputs for the memory array 33. Source linedecoder 37 in this example also decodes the output of the memory cellarray. Alternatively, bit line decoder 36 can decode the output of thememory array. The memory array serves two purposes. First, it stores theweights that will be used by the VMM. Second, the memory arrayeffectively multiplies the inputs by the weights stored in the memoryarray and adds them up per output line (source line or bit line) toproduce the output, which will be the input to the next layer or inputto the final layer. By performing the multiplication and additionfunction, the memory array negates the need for separate multiplicationand addition logic circuits and is also power efficient due to in-situmemory computation.

The output of the memory array is supplied to a differential summer(such as summing op-amp or summing current mirror) 38, which sums up theoutputs of the memory cell array to create a single value for thatconvolution. The differential summer is such as to realize summation ofpositive weight and negative weight with positive input. The summed upoutput values are then supplied to the activation function circuit 39,which rectifies the output. The activation function may include sigmoid,tanh, or ReLU functions. The rectified output values become an elementof a feature map as the next layer (C1 in the description above forexample), and are then applied to the next synapse to produce nextfeature map layer or final layer. Therefore, in this example, the memoryarray constitutes a plurality of synapses (which receive their inputsfrom the prior layer of neurons or from an input layer such as an imagedatabase), and summing op-amp 38 and activation function circuit 39constitute a plurality of neurons.

FIG. 8 is a block diagram of the various levels of VMM. As shown in FIG.8 , the input is converted from digital to analog by digital-to-analogconverter 31, and provided to input VMM 32 a. The converted analoginputs could be voltage or current. The input D/A conversion for thefirst layer could be done by using a function or a LUT (look up table)that maps the inputs to appropriate analog levels for the matrixmultiplier. The input conversion could also be done by an A/A Converterto convert an external analog input to a mapped analog input to the VMM.The output generated by the input VMM 32 a is provided as an input tothe next VMM (hidden level 1) 32 b, which in turn generates an outputthat is provided as an input to the next VMM (hidden level 2) 32 b, andso on. The various layers of VMM's 32 function as different layers ofsynapses and neurons of a convolutional neural network (CNN). Each VMMcan be a stand-alone non-volatile memory array, or multiple VMMs couldutilize different portions of the same non-volatile memory array, ormultiple VMMs could utilize overlapping portions of the samenon-volatile memory array. The example shown in FIG. 8 contains fivelayers (32 a,32 b,32 c,32 d,32 e): one input layer (32 a), two hiddenlayers (32 b,32 c), and two fully connected layers (32 d,32 e). One ofordinary skill in the art will appreciate that this is merely exemplaryand that a system instead could comprise more than two hidden layers andmore than two fully connected layers.

Vector-by-Matrix Multiplication (VMM) Arrays

FIG. 9 depicts neuron VMM 900, which is particularly suited for memorycells of the type shown in FIG. 3 , and is utilized as the synapses andparts of neurons between an input layer and the next layer. VMM 900comprises a memory array 901 of non-volatile memory cells and referencearray 902 (at the top of the array). Alternatively, another referencearray can be placed at the bottom. In VMM 900, control gates line suchas control gate line 903 run in a vertical direction (hence referencearray 902 in the row direction, orthogonal to the input control gatelines), and erase gate lines such as erase gate line 904 run in ahorizontal direction. Here, the inputs are provided on the control gatelines, and the output emerges on the source lines. In one embodimentonly even rows are used, and in another embodiment, only odd rows areused. The current placed on the source line performs a summing functionof all the currents from the memory cells connected to the source line.

As described herein for neural networks, the flash cells are preferablyconfigured to operate in sub-threshold region.

The memory cells described herein are biased in weak inversion:Ids=Io*e ^((Vg−Vth)/kVt) =w*Io*e ^((Vg)/kVt)w=e ^((−Vth)/kVt)

For an I-to-V log converter using a memory cell to convert input currentinto an input voltage:Vg=k*Vt*log[Ids/wp*Io]

For a memory array used as a vector matrix multiplier VMM, the outputcurrent is:Iout=wa*Io*e ^((Vg)/kVt), namelyIout=(wa/wp)*Iin=W*IinW=e ^((Vthp−Vtha)/kVt)

A wordline or control gate can be used as the input for the memory cellfor the input voltage.

Alternatively, the flash memory cells can be configured to operate inthe linear region:Ids=beta*(Vgs−Vth)*Vds; beta=u*Cox*W/LWα(Vgs−Vth)

For an I-to-V linear converter, a memory cell operating in the linearregion can be used to convert linearly an input/output current into aninput/output voltage.

Other embodiments for the ESF vector matrix multiplier are as describedin U.S. patent application Ser. No. 15/826,345, which is incorporated byreference herein. A sourceline or a bitline can be used as the neuronoutput (current summation output).

FIG. 10 depicts neuron VMM 1000, which is particularly suited for memorycells of the type shown in FIG. 2 , and is utilized as the synapsesbetween an input layer and the next layer. VMM 1000 comprises a memoryarray 1003 of non-volatile memory cells, reference array 1001, andreference array 1002. Reference arrays 1001 and 1002, in columndirection of the array, serve to convert current inputs flowing intoterminals BLR0-3 into voltage inputs WL0-3. In effect, the referencememory cells are diode connected through multiplexors with currentinputs flowing into them. The reference cells are tuned (e.g.,programmed) to target reference levels. The target reference levels areprovided by a reference mini-array matrix. Memory array 1003 serves twopurposes. First, it stores the weights that will be used by the VMM1000. Second, memory array 1003 effectively multiplies the inputs(current inputs provided in terminals BLR0-3; reference arrays 1001 and1002 convert these current inputs into the input voltages to supply towordlines WL0-3) by the weights stored in the memory array and then addall the results (memory cell currents) to produce the output, which willbe the input to the next layer or input to the final layer. Byperforming the multiplication and addition function, the memory arraynegates the need for separate multiplication and addition logic circuitsand is also power efficient. Here, the voltage inputs are provided onthe word lines, and the output emerges on the bit line during a read(inference) operation. The current placed on the bit line performs asumming function of all the currents from the memory cells connected tothe bitline.

FIG. 11 depicts operating voltages for VMM 1000. The columns in thetable indicate the voltages placed on word lines for selected cells,word lines for unselected cells, bit lines for selected cells, bit linesfor unselected cells, source lines for selected cells, and source linesfor unselected cells. The rows indicate the operations of read, erase,and program.

FIG. 12 depicts neuron VMM 1200, which is particularly suited for memorycells of the type shown in FIG. 2 , and is utilized as the synapses andparts of neurons between an input layer and the next layer. VMM 1200comprises a memory array 1203 of non-volatile memory cells, referencearray 1201, and reference array 1202. The reference array 1201 and 1202run in row direction of the array VMM 1200 is similar to VMM 1000 exceptthat in VMM 1200 the word lines run in the vertical direction. Here, theinputs are provided on the word lines, and the output emerges on thesource line during a read operation. The current placed on the sourceline performs a summing function of all the currents from the memorycells connected to the source line.

FIG. 13 depicts operating voltages for VMM 1200. The columns in thetable indicate the voltages placed on word lines for selected cells,word lines for unselected cells, bit lines for selected cells, bit linesfor unselected cells, source lines for selected cells, and source linesfor unselected cells. The rows indicate the operations of read, erase,and program.

FIG. 14 depicts neuron VMM 1400, which is particularly suited for memorycells of the type shown in FIG. 3 , and is utilized as the synapses andparts of neurons between an input layer and the next layer. VMM 1400comprises a memory array 1403 of non-volatile memory cells, referencearray 1401, and reference array 1402. The reference array 1401 and 1402serves to convert current inputs flowing into terminals BLR0-3 intovoltage inputs CG0-3. In effect, the reference memory cells are diodeconnected through cascoding mulitplexors 1414 with current inputsflowing into them. The mux 1414 includes a mux 1405 and a cascodingtransistor 1404 to ensure a constant voltage on bitline of referencecells in read. The reference cells are tuned to target reference levels.Memory array 1403 serves two purposes. First, it stores the weights thatwill be used by the VMM 1400. Second, memory array 1403 effectivelymultiplies the inputs (current inputs provided to terminals BLR0-3;reference arrays 1401 and 1402 convert these current inputs into theinput voltages to supply to the control gates CG0-3) by the weightsstored in the memory array and then add all the results (cell currents)to produce the output, which will be the input to the next layer orinput to the final layer. By performing the multiplication and additionfunction, the memory array negates the need for separate multiplicationand addition logic circuits and is also power efficient. Here, theinputs are provided on the word lines, and the output emerges on thebitline during a read operation. The current placed on the bitlineperforms a summing function of all the currents from the memory cellsconnected to the bitline.

VMM 1400 implements uni-directional tuning for memory cells in memoryarray 1403. That is, each cell is erased and then partially programmeduntil the desired charge on the floating gate is reached. If too muchcharge is placed on the floating gate (such that the wrong value isstored in the cell), the cell must be erased and the sequence of partialprogramming operations must start over. As shown, two rows sharing thesame erase gate need to be erased together (to be known as a pageerase), and thereafter, each cell is partially programmed until thedesired charge on the floating gate is reached,

FIG. 15 depicts operating voltages for VMM 1400. The columns in thetable indicate the voltages placed on word lines for selected cells,word lines for unselected cells, bit lines for selected cells, bit linesfor unselected cells, control gates for selected cells, control gatesfor unselected cells in the same sector as the selected cells, controlgates for unselected cells in a different sector than the selectedcells, erase gates for selected cells, erase gates for unselected cells,source lines for selected cells, and source lines for unselected cells.The rows indicate the operations of read, erase, and program.

FIG. 16 depicts neuron VMM 1600, which is particularly suited for memorycells of the type shown in FIG. 3 , and is utilized as the synapses andparts of neurons between an input layer and the next layer. VMM 1600comprises a memory array 1603 of non-volatile memory cells, referencearray 1601, and reference array 1602. EG lines are run vertically whileCG and SL lines are run horizontally. VMM 1600 is similar to VMM 1400,except that VMM 1600 implements bi-directional tuning, where eachindividual cell can be completely erased, partially programmed, andpartially erased as needed to reach the desired amount of charge on thefloating gate. As shown, reference arrays 1601 and 1602 convert inputcurrent in the terminal BLR0-3 into control gate voltages CG0-3 (throughthe action of diode-connected reference cells through multiplexors) tobe applied to the memory cells in the row direction. The current output(neuron) is in the bitline which sums all currents from the memory cellsconnected to the bitline.

FIG. 17 depicts operating voltages for VMM 1600. The columns in thetable indicate the voltages placed on word lines for selected cells,word lines for unselected cells, bit lines for selected cells, bit linesfor unselected cells, control gates for selected cells, control gatesfor unselected cells in the same sector as the selected cells, controlgates for unselected cells in a different sector than the selectedcells, erase gates for selected cells, erase gates for unselected cells,source lines for selected cells, and source lines for unselected cells.The rows indicate the operations of read, erase, and program.

The prior art includes a concept known as long short-term memory (LSTM).LSTM units often are used in neural networks. LSTM allows a neuralnetwork to remember information over arbitrary time intervals and to usethat information in subsequent operations. A conventional LSTM unitcomprises a cell, an input gate, an output gate, and a forget gate. Thethree gates regulate the flow of information into and out of the cell.VMMs are particularly useful in LSTM units.

FIG. 28 depicts exemplary LSTM 2800. LSTM in this example comprisescells 2801, 2802, 2803, and 2804. Cell 2801 receives input vector x₀ andgenerates output vector h₀ and cell state vector c₀. Cell 2802 receivesinput vector x₁, the output vector (hidden state) h₀, and cell state c₀from cell 2801 and generates output vector h₁ and cell state vector Cell2803 receives input vector x₂, the output vector (hidden state) h₁ andcell state c₁ from cell 2802 and generates output vector h₂ and cellstate vector c₂. Cell 2804 receives input vector x₃, the output vector(hidden state) h₂, and cell state c₂ from cell 2803 and generates outputvector h3. Additional cells can be used, and an LSTM with four cells ismerely an example.

FIG. 29 depicts an exemplary implementation of LSTM cell 2900, which canbe used for cells 2801, 2802, 2803, and 2804 in FIG. 28 . LSTM cell 2900receives input vector x(t) and cell state vector c(t−1) from a precedingcell and generates cell state(t) and output vector h(t).

LSTM cell 2900 comprises sigmoid function devices 2901, 2902, and 2903,each of which applies a number between 0 and 1 to control how much ofeach component in the input vector is allowed through to the outputvector. LSTM cell 2900 also comprises tank devices 2904 and 2905 toapply a hyperbolic tangent function to an input vector, multiplierdevices 2906, 2907, and 2908 to multiply two vectors together, andaddition device 2909 to add two vectors together.

FIG. 30 depicts LSTM cell 3000, which is an example of an implementationof LSTM cell 2900. For the reader's convenience, the same numbering fromFIG. 29 and LSTM cell 2900 is used in FIG. 30 and LSTM cell 3000. As canbe seen in FIG. 30 , sigmoid function devices 2901, 2902, and 7903 andtank devices 2904 and 2905 each comprise multiple VMM arrays 3001. Thus,it can be seen that VMM arrays are particular important in LSTM cellsused in certain neural network systems.

It can be further appreciated that LSTM systems will typically comprisemultiple VMM arrays, each of which requires functionality provided bycertain circuit blocks outside of the VMM arrays, such as a summer andactivation circuit block and high voltage generation blocks. Providingseparate circuit blocks for each VMM array would require a significantamount of space within the semiconductor device and would be somewhatinefficient. The embodiments described below therefore attempt tominimize the circuitry required outside of the VMM arrays themselves.

Similarly, an analog VMM implementation can be used for a GRU (gatedrecurrent unit) system. GRUs are a gating mechanism in recurrent neuralnetworks. GRUs are similar to LSTMs, with one notable difference beingthat GRUs lack an output gate.

FIG. 31 depicts exemplary GRU 3100. GRU in this example comprises cells3101, 3102, 3103, and 3104. Cell 3101 receives input vector x₀ andgenerates output vector h₀ and cell state vector c₀. Cell 3102 receivesinput vector x₁, the output vector (hidden state) h₀, and cell state c₀from cell 3101 and generates output vector h₁ and cell state vector c₁.Cell 3103 receives input vector x₂, the output vector (hidden state) h₁,and cell state c₁ from cell 3102 and generates output vector h₂ and cellstate vector c₂. Cell 3104 receives input vector x₃, the output vector(hidden state) h₂, and cell state c₂, from cell 3103 and generatesoutput vector h3. Additional cells can be used, and an GRU with fourcells is merely an example.

FIG. 32 depicts an exemplary implementation of GRU cell 3200, which canbe used for cells 3101, 3102, 3103, and 3104 in FIG. 31 . GRU cell 3200receives input vector x(t) and cell state vector h(t−1) from a precedingcell and generates cell state h(t). GRU cell 3200 comprises sigmoidfunction devices 3201 and 3202 each of which applies a number between 0and 1 to components from cell state h(t−1) and input vector x(t). GRUcell 3200 also comprises tank device 3203 to apply a hyperbolic gfunction to a input vector, multiplier devices 3204, 3205, and 3206 tomultiply two vectors together, addition device 3207 to add two vectorstogether, and complementary device 3208 to subtract an input from 1 togenerate an output.

FIG. 33 depicts GRU cell 3300, which is an example of an implementationof GRU cell 3200. For the reader's convenience, the same numbering fromFIG. 32 and GRU cell 3200 is used in FIG. 33 and GRU cell 3300. As canbe seen in FIG. 33 , sigmoid function devices 3201 and 3202, and tanhdevice 3203 each comprise multiple VMM arrays 3301. Thus it can be seenthat VMM arrays are of particular use in GRU cells used in certainneural network systems.

It can be further appreciated that GRU systems will typically comprisemultiple VMM arrays, each of which requires functionality provided bycertain circuit blocks outside of the VMM arrays, such as a summer andactivation circuit block and high voltage generation blocks. Providingseparate circuit blocks for each VMM array would require a significantamount of space within the semiconductor device and would be somewhatinefficient. The embodiments described below therefore attempt tominimize the circuitry equired outside of the VMM arrays themselves.

FIG. 18 depicts VMM system 1800. VMM system 1800 comprises VMM array1807, low voltage row decoder 1803, high voltage row decoder 1805,reference cell low voltage column decoder 1806 (shown for the referencearray in the column direction, meaning providing input to outputconversion in the row direction), bit line PE driver 1802, bit linemultiplexor 1808, activation function circuit and summer 1809, controllogic 1804, and analog bias circuit 1801.

Low voltage row decoder 1803 provides a bias voltage for read andprogram operations and provides a decoding signal for high voltage rowdecoder 1805. High voltage row decoder 1805 provides a high voltage biassignal for program and erase operations. Bit line PE driver 1801provides controlling function for bit line in program, verify, anderase. Bias circuit 1801 is a shared bias block that provides themultiple voltages needed for the various program, erase, program verify,and read operations.

VMM system 1800 further comprises redundancy array 1810 and/orredundancy array 1813. Redundancy arrays 1810 and 1813 each providearray redundancy for replacing a defective portion in array 1807, inaccordance with the redundancy embodiments described in greater detailbelow.

VMM system 1800 further comprises NVR (non-volatile register, aka infosector) sectors 1811, which are array sectors used to store user info,device ID, password, security key, trimbits, configuration bits,manufacturing info, etc.

VMM system 1800 optionally comprises reference sector 1812 and/orreference system 1850. Reference system 1850 comprises reference array1852, reference array low voltage row decoder 1851, reference array highvoltage row decoder 1853, and reference array low voltage column decoder1854. The reference system can be shared across multiple VMM systems.

Reference array low voltage row decoder 1851 provides a bias voltage forread and programming operations involving reference array 1852 and alsoprovides a decoding signal for reference array high voltage row decoder1853. Reference array high voltage row decoder 1853 provides a highvoltage bias for program and operations involving reference array 1852.Reference array low voltage column decoder 1854 provides a decodingfunction for reference array 1852. Reference array 1852 is such as toprovide reference target for program verify or cell margining (searchingfor marginal cells).

FIG. 19 depicts prior art memory system 1900 that provides redundancyfor rows containing one or more faulty memory cells during program,erase, or read operations. Memory system 1900 comprises addresscomparator 1903, logic circuit 1905, array 1906, redundancy array 1907,row decoder 1908, and redundancy row decoder 1909.

During a testing or configuration phase, each row of memory cells inarray 1906 is tested and verified. Any memory cells deemed to be faultyare identified, and the address for each row that contains one or morefaulty memory cells is stored in non-volatile memory (not shown).Thereafter, during operation of memory system 1900, each address 1901for a read or write operation is compared by address comparator 1903against each address in the set of stored addresses corresponding torows containing one or more faulty memory cells.

If a match is found by address comparator 1903 with any of the storedaddresses, enable signal 1904 is asserted, which signifies that thereceived address is for a faulty row. Enable signal 1904 is received byredundant array row decoder 1909, which then selects a row that has beenassigned to the row containing the faulty memory cell. Thus, theprogram, erase, or read operation is directed to the redundant rowinstead of the row containing the faulty memory cell.

If a match is not found by address comparator 1903, then enable signal1904 is de-asserted, and row decoder 1908 is enabled by the output oflogic circuit 1905 (here shown as an inverter). In this situation, thereceived address 1901 is used to access a row in array 1906 for theoperation.

By design, only one of the assigned redundant rows can be asserted atany given time. This prior art system therefore could not be used toperform a neural read operation, where all non-faulty rows and assignedredundant rows are asserted.

FIG. 20 depicts an embodiment of an improved memory system. Memorysystem 2000 provides redundancy for rows containing one or more badmemory cells. Unlike memory system 1900, memory system 2000 is able toassert all assigned redundant rows during a neural read operation, andit also can perform a program, erase, or read operation as in memorysystem 1900. Memory system 2000 contains the same components as memorysystem 1900 except that row decoder and redundancy latch block 2008 isused instead of row decoder 1908. Row decoder and redundancy latch block2008 contains circuitry that enables all assigned redundant rows to beasserted during a neural read operation.

FIG. 21 depicts further detail regarding an embodiment of row decoderand redundancy latch block 2008, here shown as row decoder andredundancy latch block 2101. Memory system 2100 comprises row decoderand redundancy latch block 2101, array 2102, redundancy array 2103, andhigh voltage decoder 2104. Row decoder and redundancy latch block 2101comprises numerous instances of sub-block 2105, where each instance ofsub-block 2105 is coupled to a pair of word lines in array 2102 (here,word lines 2115 and 2116). Thus, in this embodiment, sub-block 2105 andsimilar sub-blocks each are coupled to a sector of memory in array 2102.In an alternative embodiment, each sub-clock can be coupled to more thantwo rows.

Row decoder and redundancy latch block 2101 further comprises redundancysub-block 2113 coupled to a pair of word lines in redundancy array 2103(here, word lines 2117 and 2118). Additional redundancy sub-blockssimilar to redundancy sub-block 2113 can be included in row decoder andredundancy latch block 2101.

Sub-block 2105 comprises NAND gates 2106 and 2107, inverters 2108 and2109, NAND gates 2110 and 2111, inverters 2112 and 2113, and latch 2114.Latch 2114 is programmed (or loaded with configuration data at power-upor in response to a redundancy load command) during a testing orconfiguration phase. If word lines 2115 or 2116 are coupled to a rowcontaining one or more faulty memory cells, then a “1” will beprogrammed into latch 2114. Otherwise, latch 2114 will store a “0”.During normal operation, when NAND gates 2106 or 2107 receive an addresscorresponding to word lines 2115 or 2116, respectively, latch 2114 willcause that word line to be de-asserted instead of asserted. Thus, theword line containing the faulty memory cell will not be selected.

Redundancy sub-block 2113 contains similar components as sub-block 2105.Here, latch 2119 is programmed during a testing or configuration phase.If word lines 2117 or 2118 are to be used, then latch 2119 is programmedwith a “0”. Otherwise, latch 2119 is programmed with a “1”. Duringnormal operation, when the receiving NAND gate receives an addresscorresponding to word lines 2117 or 2118, latch 2119 will cause thatword line to be asserted. Thus, the word line containing the redundantmemory cells will be selected. Notably, multiple redundant rows can beselected at any given time (such as during a neural read operation) byconfiguring latches such as latch 2119 with a “0”.

FIG. 22 depicts further detail regarding another embodiment of rowdecoder and redundancy latch block 2008, here shown as row decoder andredundancy latch block 2201. Memory system 2200 comprises row decoderand redundancy latch block 2201, array 2202, redundancy array 2203, andhigh voltage decoder 2204. Row decoder and redundancy latch block 2201contains numerous instances of sub-block 2205, where each instance ofsub-block 2205 is coupled to a word line in array 2202 (here, word line2211). Thus, in this embodiment, sub-block 2205 and similar sub-blockseach are coupled to a sector of memory in array 2202.

Row decoder and redundancy latch block 2201 further comprises redundancyblock 2213 coupled to a redundant word line in redundancy array 2203(here, word line 2212).

Sub-block 2205 comprises NAND gate 2206, inverter 2207, latch 2208, andswitches 2209 and 2210. Here, redundancy latch 2208 is programmed duringa configuration stage of memory system 2200. If latch 2208 contains a“1”, then the corresponding row coupled to word line 2211 in array 2202is not faulty. During normal operation, switch 2209 will be closed andswitch 2210 will be open, and word line 2211 in array 2202 will beaccessed when the appropriate address is received. If latch 2208contains a “0”, then the corresponding row in array 2202 is faulty.During normal operation, switch 2209 will be open and switch 2109 willbe closed, and word line 2211 in array 2202 will not be accessed whenthe appropriate address is received. Instead, redundant word line 2212in array 2202 will be accessed through a vertical line 2215.

FIG. 23 depicts another embodiment of the inventive concepts. Memorysystem 2300 comprises column decoder and redundancy latch block 2301,array 2302, and redundancy array 2303. Column decoder and redundancylatch block 2301 contains numerous instances of sub-block 2304, whereeach instance of sub-block 2304 is selectively coupled to a bit line ora group of bitlines (here, bit line 2308) in array 2302. Sub-block 2304comprises latch 2305, switch 2306, and switch 2307.

Latch 2305 is programmed during a testing or configuration stage ofmemory system 2300. If latch 2305 contains a “1”, then the correspondingcolumn coupled to bit line 2308 is not faulty. During normal operation,switch 2306 will be closed and switch 2307 will be open, and bit line2308 will be accessed when the appropriate address is received. If latch2305 contains a “0”, then the corresponding column in array 2302 isfaulty. During normal operation, switch 2306 will be open and switch2307 will be closed, and bit line 2308 in array 2202 will not beaccessed when the appropriate address is received. Instead, redundantbit line 2309 in redundancy array 2303 will be accessed through ahorizontal line 2315.

FIG. 24 depicts program, program verify, and erase method 2400. Theprocess starts (step 2401). A program, program verify, or erase commandis received (step 2402). The received address is compared against theaddresses for known bad rows or columns in a memory array (step 2403).

If a match is found (step 2404), that means a bad address exists. Thesystem will disable the bad address (step 2405) and enable acorresponding redundant address (step 2406). The program, programverify, or erase command is then executed using the redundant address(step 2407). The process is then complete (step 2408).

If a match is not found (step 2404), the received address is enabled(step 2409). The program, program verify, or erase command is thenexecuted using the received address (step 2410). The process is thencomplete (step 2408).

FIG. 25 depicts an embodiment for neural read process 2500. The processstarts (step 2501). Redundancy latches are loaded or configured withredundancy information (step 2502). A neural read operation occurs,whereby the entire array and redundancy array are enabled except for thebad rows or columns (step 2503). The process is them complete (step2504).

FIG. 26 depicts decoding circuitry 2600 that is suitable for use witharrays containing memory cells of the type shown in FIG. 2 . Decodingcircuitry 2600 comprises word line decoder 2601, high voltage supply2604, and source line decoder 2606. Word line decoder 2601 comprisesPMOS transistor 2603 and NMOS transistor 2602, configured as shown.Source line decoder 2606 comprises NMOS transistors 2607, 2608, and2609, configured as shown. High voltage supply 2604 comprises highvoltage logic supply 2605.

FIG. 27 depicts decoding circuitry 2700 that is suitable for use witharrays containing memory cells of the type shown in FIG. 3 . Decodingcircuitry 2700 comprises erase gate line decoder 2701, control gatedecoder 2706, source line decoder 2709, and high voltage supply 2704.Erase gate decoder 2701 comprises PMOS transistor 2703 and NMOStransistor 2702, configured as shown. Control gate decoder 2706comprises PMOS transistor 2708 and NMOS transistor 2707. Source linedecoder 2709 comprises NMOS transistors 2710, 2711, and 2712, configuredas shown. High voltage supply 2704 comprises high voltage logic supply2705.

It should be noted that, as used herein, the terms “over” and “on” bothinclusively include “directly on” (no intermediate materials, elementsor space disposed therebetween) and “indirectly on” (intermediatematerials, elements or space disposed therebetween). Likewise, the term“adjacent” includes “directly adjacent” (no intermediate materials,elements or space disposed therebetween) and “indirectly adjacent”(intermediate materials, elements or space disposed there between),“mounted to” includes “directly mounted to” (no intermediate materials,elements or space disposed there between) and “indirectly mounted to”(intermediate materials, elements or spaced disposed there between), and“electrically coupled” includes “directly electrically coupled to” (nointermediate materials or elements there between that electricallyconnect the elements together) and “indirectly electrically coupled to”(intermediate materials or elements there between that electricallyconnect the elements together). For example, forming an element “over asubstrate” can include forming the element directly on the substratewith no intermediate materials/elements therebetween, as well as formingthe element indirectly on the substrate with one or more intermediatematerials/elements there between.

What is claimed is:
 1. A method of performing a neural read operation ina memory system comprising a memory array and a redundant memory array,the method comprising: loading data into one or more latches; disablinga plurality of rows of memory cells in the memory array in response tothe one or more latches; enabling a plurality of rows of memory cells inthe redundant memory array; and performing a concurrent read operationof all memory cells in non-disabled rows in the memory array and allmemory cells in the plurality of enabled rows in the redundant memoryarray.
 2. The method of claim 1, further comprising: receiving a currenton each output line in the memory array and redundant memory array,wherein the current on each output line comprises current drawn duringthe concurrent read operation by each memory cell in a non-disabled rowin the memory array coupled to the output line and each memory cell inthe plurality of enabled rows in the redundant memory array coupled tothe output line.
 3. The method in claim 2, wherein the output line is abitline.
 4. The method in claim 2, wherein the output line is a sourceline.
 5. The method of claim 1, wherein the disabling utilizes discretelogic.
 6. The method of claim 1, wherein the disabling utilizes one ormore switches.
 7. The method of claim 1, wherein each of the one or morelatches is coupled to word lines for a sector of memory cells in thememory array.
 8. The method of claim 1, wherein each of the memory cellsin the memory array and each of the memory cells in the redundant memoryarray is a split-gate flash memory cell.
 9. The method of claim 1,further comprising performing an address comparison during a programmingor erasing operation to determine if an address corresponds to faultymemory.
 10. The method of claim 9, wherein the redundant memory array isenabled for a program or erase operation if an address comparisonidentifies a match.
 11. A method of performing a neural read operationin a memory system comprising a memory array and a redundant memoryarray, the method comprising: loading data into one or more latches;disabling a plurality of columns of memory cells in the memory array inresponse to the one or more latches; enabling a plurality of columns ofmemory cells in the redundant memory array; and performing a concurrentread operation of all memory cells in non-disabled columns in the memoryarray and all memory cells in the plurality of enabled columns in theredundant memory array.
 12. The method of claim 11, further comprising:receiving a current on each output line in the memory array andredundant memory array, wherein the current on each output linecomprises current drawn during the read operation by each memory cell ina non-disabled column in the memory array coupled to the output line andeach memory cell in an enabled column in the redundant memory arraycoupled to the output line.
 13. The method of claim 12, wherein theoutput line is a bitline.
 14. The method of claim 12, wherein the outputline is a source line.
 15. The method of claim 11, wherein the disablingis performed by one or more switches.
 16. The method of claim 11,wherein each of the memory cells in the memory array and each of thememory cells in the redundant memory array is a split-gate flash memorycell.
 17. The method of claim 11, further comprising performing anaddress comparison during a programming or erasing operation todetermine if an address corresponds to faulty memory.
 18. The method ofclaim 17, wherein the redundant memory array is enabled for a program orerase operation if an address comparison identifies a match.
 19. Amemory system comprising: a memory array comprising a plurality ofsectors, each sector comprising a plurality of rows of memory cells; aredundant memory array comprising a plurality of redundant sectors, eachredundant sector comprising a plurality of rows of memory cells; foreach sector in the memory array, a control block comprising a latch,wherein the latch can be programmed to disable one or more rows in thesector in the memory array; and for each sector in the redundant memoryarray, a control block comprising a redundancy latch, wherein theredundancy latch can be programmed to enable one or more rows in aredundant sector in the redundant memory array; wherein, during a neuralread operation, a concurrent read operation is performed of all memorycells in non-disabled rows in the memory array and all memory cells inthe enabled rows in the redundant memory array.
 20. The system of claim19, further comprising: circuitry for receiving a current on each bitline in the memory array and redundant memory array, wherein the currenton each bit line comprises current drawn during the read operation byeach memory cell in a non-disabled row in the memory array coupled tothe bit line and each memory cell in an enabled row in the redundantmemory cell coupled to the bit line.
 21. The system of claim 19, furthercomprising: for each sector in the memory array, discrete logic betweenthe latch and the memory array.
 22. The system of claim 19, furthercomprising: for each sector in the memory array, one or more switchesbetween the latch and the memory array.
 23. The system of claim 19,wherein each of the memory cells in the memory array and each of thememory cells in the redundant memory array is a split-gate flash memorycell.
 24. The system of claim 19, wherein the memory array is avector-by-matrix multiplication array in a long short term memorysystem.
 25. The system of claim 19, wherein the memory array is avector-by-matrix multiplication array in a gated recurrent unit system.26. A memory system comprising: a memory array comprising memory cellsarranged in rows and columns, wherein each column of memory cells iscoupled to a bit line; a redundant memory array comprising redundantmemory cells arranged in rows and columns, wherein each column ofredundant memory cells is coupled to a bit line; for each bit line inthe memory array, a control block comprising a latch, wherein the latchcan be programmed to disable the column of memory cells coupled to thebit line in the memory array; and for each bit line in the redundantmemory array, a control block comprising a redundancy latch, wherein theredundancy latch can be programmed to enable a column of memory cellscoupled to the bit line in the redundant memory array; wherein, during aneural read operation, a concurrent read operation is performed of allmemory cells in non-disabled columns in the memory array and all memorycells in the enabled columns in the redundant memory array.
 27. Thesystem of claim 26, further comprising: circuitry for receiving acurrent on each bit line coupled to a non-disabled column in the memoryarray and each bit line coupled to an enabled column in the redundantmemory array during a neural read operation.
 28. The system of claim 26,further comprising: for each bit line in the memory array, one or moreswitches between the latch and the memory array.
 29. The system of claim26, wherein each of the memory cells in the memory array and each of thememory cells in the redundant memory array is a split-gate flash memorycell.
 30. The system of claim 26, wherein the memory array is avector-by-matrix multiplication array in a long short term memorysystem.
 31. The system of claim 26, wherein the memory array is avector-by-matrix multiplication array in a gated recurrent unit system.